Can Modeling Redundancy In Multimodal, Multi-party Tasks Support Dynamic Learning?

In multi-party interactions humans use available communication modes in predictable ways. For example, the dialogue theories of Conversational Implicature (Grice 1975) and Giveness Theory (Gundel, Hedberg et al. 1993) have both been applied successfully in the analysis of multi-party, multimodal settings (Chai, Prasov et al. 2005). At times people use multiple modes of communication in complementary or mutually disambiguating ways (Oviatt and Olsen 1994), while at other times the information in multiple modes is redundant (Anderson, Hoyer et al. 2004) (Fig. 2). Our position is that gaining a better understanding of why, when and how people choose to communicate multimodal information redundantly is very important for emerging computational systems that aim to be intelligent assistants for humans. Our technique of Multimodal New Vocabulary Recognition (MNVR) learns the spelling, pronunciation and semantics of new, out-of-vocabulary (OOV) words from a single observation of redundant handwriting and speech in a naturally occurring exchange of information within a multi-party scheduling meeting (Fig. 1). Similar redundancies can occur across other modes (e.g. gazing at someone while speaking their name). We believe that empirical research into the nature of communicative redundancy could be a very informative guide to the development and integration of a generalized dynamic learning approach in evolving multimodal interfaces.